...while for B, "2" doesn't really mean anything; it's just a symbol that it blindly manipulates.

I think I understand what concepts you were gesturing towards with this example, but for me the argument doesn't go through. The communication failure suggests to me that you might need to dissolve some questions around syntax, semantics, and human pscyhology. In the absence of clear understanding here I would fear a fallacy of equivocation on the term "meaning" in other contexts as well.

The problem is that B seems to output a "3" every single time it sees a "2" in the input. By "3" it functionally communicates "there was a '2' in the corresponding input" and presumably a "2" in the output functionally communicates some stable fact of the input such as that there was a "Q" in it.

This is a different functional meaning that A communicates, but the distance between algorithms A and B isn't very far. One involves a little bit of code and the other involves a little bit more, but these are both relatively small scripts that can be computed using little more memory than is needed to store the input itself.

I could understand someone using the term "meaning" to capture a sense where A and B are both capable of meaning things because they functionally communicate something to a human observer by virtue of their stably predictable input/output relations. Equally, however, I would accept a sense where neither algorithm was capable of meaning something because (to take one trivial example of the way humans are able to "mean" or "not mean" something) neither algorithm is capable of internally determining the correct response, examining the speech context they find themselves in, and emitting either the correct response or a false response that better achieves their goals within that speech context (such as to elicit laughter or to deceive the listener).

You can't properly ask algorithm A "Did you really mean that output?" and get back a sensible answer, because algorithm A has no English parsing abilities, nor a time varying internal state, nor social modeling processes capable of internally representing your understanding (or lack of understanding) of its own output, nor a compressed internal representation of goal outcomes (where fiddling with bits of the goal representation would leave algorithm A continuing to produce complex goal directed behavior except re-targeted at some other goal than the outcome it was aiming for before its goal representation was fiddled with).

I'd be willing to accept an argument that used "meaning" in a very mechanistic sense of "reliable indication" or a social sense of "honest communication where dishonest communication was possible" or even some other sense that you wanted to spell out and then re-use in a careful way in other arguments... But if you want to use a primitive sense of "meaning" that applies to a calculator and then claim that that is what I do when I think or speak, then I don't think I'll find it very convincing.

My understanding of words like "meaning" and conjugations of "to be" starts from the assumption that they are levers for referring to the surface layer of enormously complex cognitive modeling tools for handling many radically different kinds of phenomena where it is convenient to paper over the complexity to order to get some job done, like dissecting precisely what a love interest "meant" when they said you "were being" coy. What "means" means, or what "were being" means in that sort of context is patently obvious to your average 13 year old... except that its really hard to spell that sort of thing out precisely enough to re-implement it in code or express the veridicality conditions in formal logic over primitive observables.

We are built to model minds, just like we are built to detect visual edges. We do these tasks wonderfully and we introspect on them terribly, which means re-using a concept like "meaning" in your foundational moral philosophy is asking for trouble :-P

I normally think of meaning in terms of isomorphisms between systems.

That is, if we characterize a subset of domain X as entity Ax with relationships to other entities in X (Bx, Cx, and so forth), and we characterize those relationships in certain ways... inhibitory and excitory linkages of spreading activation, for example, or correlated appearance and disappearance, or identity of certain attributes like color or size, and entity Ay in domain Y has relationships to other entities in Y (By, Cy, and so forth), and there's a way to characterize Ax's relationships to Bx, Cx, etc. such that Ay has the same relationships to By, Cy, etc., then it makes sense to talk about Ax, Bx, and Cx having the same meaning as Ay, By, and Cy.

(In and of itself, this is a symmetrical relationship... it suggests that if "rock" means rock, then rock means "rock." Asymmetry can be introduced via the process that maintains the similar relationships and what direction causality flows through it... for example, if saying "The rock crumbles into dust" causes the rock to crumble into dust, it makes sense to talk about the object meaning the word; if crumbling the rock causes me to make that utterance, it makes sense to talk about the word meaning the object. If neither of those things happens, the isomorphism is broken and it stops making sense to say there's any meaning involved at all. But I digress.)

The relationship between the string "2" in a calculator and the number of FOOs in a pile of FOOs (or, to state that more generally, between "2" and the number 2) is different for A and B, so "2" means different things in A and B.

But it's worth noting that isomorphisms can be discovered by adopting new ways of characterizing relationships and entities within domains. That is, the full meaning of "2" isn't necessarily obvious, even if I've been using the calculator for a while. I might suddenly discover a way of characterizing the relationships in A and B such that "2" in A and B are isomorphic... that is, I might discover that "2" really does mean the same thing in A and B after all!

This might or might not be a useful discovery, depending on how useful that new way of characterizing relationships is.

As lukeprog notes, all of this is pretty standard philosophy of language.

So, if asked what "right" means to me, I'm ultimately inclined to look at what relationship "right" has to other things in my head, and what kinds of systems in the world have isomorphic patterns of relationships, and what entities in those systems correspond to "right", and whether other people's behavior is consistent with their having similar relationships in their head.

I mostly conclude based on this exercise that to describe an action as being "right" is to imply that a legitimate (though unspecified) authority endorses that action. I find that increasingly distasteful, and prefer to talk about endorsing the action myself.

We could perhaps say that by "2" B means "the way B manipulates the symbol '2'", but that doesn't seem to buy us anything.

I think the meaning of a concept-detector (the concept it detects) in the more naive sense should be considered in the context of an agent, and refers to the way this concept-detector should be, as far as the agent can see (i.e. it's a question of knowable self-improvement). And the way it should be of course depends on how its properties control agent's actions, i.e. on the role the symbol plays in agent's algorithm. (This promises simplification in so far as the context-detector is blind to its purpose, i.e. its usability and direction of refinement is seen only in terms of other instrumental heuristics.)

You cannot make sense of "meaning" unless you assume some form of compositionality. We know the meaning of A: it is a (partial) function which maps some strings as input to some other strings as output. We could decompose A into an algorithm which parses a string to an abstract syntax tree, another algorithm which evaluates the AST and outputs an internal representation of its result, and a third algorithm which prettyprints the result. This makes sense, since our programming language probably has a syntax construct with the meaning of function composition.

But then what's the meaning of an input, such as "2" ? Taking our algorithm as given, "2" is sort of like a function which waits for the rest of our input and then feeds it to the evaluator. Under some niceness assumptions, we can reason about the whole thing and get something better: "2" means: "the leftmost leaf of the abstract syntax tree is of the form: 2 * 10^n + i_(n-1) * 10^(n-1) + i_(n-2) * 10^(n-2) + ... + i_1 * 10 + i_0 , where n >= 0 and each i_k is a natural number from 0 to 9". This assumes that we can assign natural numbers and trees as the meaning of our inner representation, which requires some handwaving in itself. If the programming language has primitive data types and operations for these, then this is easily justified. Otherwise, we would need to prove that some data representations within our program are isomorphic to the naturals, etc. and that the operations are also implemented correctly.

Is it more like calculator A or B?

As a layman I'm not sure how B is not arithmetic as well?

...then the way that a human mind processes such arguments can be viewed as an algorithm...

What is right isn't defined by the algorithm alone but also by the environment, nurture, education and a lot of other variables. Change the circumstances and you change what is right.

You might argue that there are arguments that do not satisfy our evolutionary mind template. But that's not true, because we already encounter rules that don't feel intuitively right but that we nonetheless adopt. In such situations we call ourselves biased and want to change our minds. That's why an FAI trying to extrapolate human volition might fail for two reasons: 1.) our volition is broad and fuzzy, to an extent that right is too vague and volatile 2.) by changing the circumstances it will change what is right and therefore eventually define what is right.