Htydjtk

Do we know what the symbol is that Gandalf puts on the door of Bag End?

I mean the one in the novel. Is it the same as the one in the movie?

About the symbol/letter (Theory A)

@Jimmy Shelter's answer is the one that should be marked as the correct (as it answers the author's doubt), but as a comment I'd like to talk about the symbol itself.

At first sight I thought it was just an "F" (feoh) in Futhorc. Futhorc is an old Anglo-Saxon runic alphabet, that Tolkien uses in the maps we can see in the book.

Later on, in the Lord of the Rings, Tolkien uses Cirth, an invented alphabet based on Futhorc (and futhark, and similars), to write Khuzdul: the language of the Dwarrow. Probably the best example for this is Balin's tomb in the Fellowship of the Ring.

BALIN FUNDINUL UZBADKHAZADDUMU BALINSONOVFUNDINLORDOVMORIA (Balin
Fundinul Uzbad Khazaddumu in Khuzdul - Balin Son of Fundin Lord of
Moria in Westron [English]).

I thought they would use Cirth in the Hobbit films, however, every piece of Khuzdul (English, actually) is written using Futhorc (click on the images to enlarge them):

"Stand by the grey stone when the thrush knocks and the setting sun
with the last light of Durin's day will shine upon the keyhole. Th."
(in perfect English).

"Herein lies the seventh kingdom of Durin's folk. If this fortress be
lost or overthrown, look to the power of the Arkenstone." (again, in
perfect English).

This is actually weird, as in the movie Glóin reads it:

"Herein lies the seventh kingdom of Durin’s folk. May the Heart of
the Mountain unite all Dwarves in defence of this home."

As I said before, I expected every map and so on written in Khuzdul using Cirth, but I appreciate the fact that Jackson and the artistic team have respected the fact that Cirth didn't exist at the time the Hobbit was created.

Having said that, the only conclusion is that the letter is just an English F. I don't know how that relates to "[...] the usual one in the trade, or used to be.", but that's the way it is. The only thing that comes to my mind is that a synonym to trade or maybe a related word starts with F in Ænglisc.

Theory B

Some say it's a G in Cirth that somehow connects to the G (Gandalf) seen in Gandalf's carriage (The Fellowship of the Ring), but in my honest opinion that makes no sense in the context of the film and the story: why would Gandalf mark Bilbo's door with his very name?

This seems to be the correct theory, apparently (see the bottom of the answer).

What should have been done

The illustration above was made by Master Tolkien himself. It depicts the letters B, D and a symbol (probably a diamond). Now, let's discuss their meaning:

“Yes, yes, but that was long ago,” said Glóin. “I was talking about you. And I assure you there is a mark on this door—the usual one in the trade, or used to be. Burglar wants a good job, plenty of Excitement and reasonable Reward, that’s how it is usually read.

_{The Hobbit, Chapter I: An Unexpected Party.}

Thanks to this quote we may affirm the symbols mean: Burglar, Danger (excitement), and Reward (a diamond).

Actual meaning in the Jacksonverse

Why does the film show a Futhorc F (or a Cirth G)? Most likely because of the timing of the scene: what works in books doesn't necessary work in a motion picture.

UPDATE:

All seems to indicate that the in-universe explanation is Theory B, as Weta Workshop sells a Stone Pendant collectible with the Feoh (Cirth 19, actually) letter under the name of The Mark of Gandalf. It's worth noting that it clearly is a The Hobbit collectible and not a The Lord of the Rings one, as you can read from the watermark on the picture below.

The actual form of the symbol is not described in the book; the most we get is that Gandalf "scratched a queer sign" on the door:

After a while he stepped up, and with the spike of his staff scratched a queer sign on the hobbit's beautiful green front-door.

Later on Gloin describes what the sign means:

In fact, if it had not been for the sign on the door, I should have been sure we had come to the wrong house ... I assure you there is a mark on this door - the usual one in the trade, or used to be. Burglar wants a good job, plenty of excitement and reasonable reward, that's how it is usually read.

And Gandalf had also taken steps to cover his tracks, which seems rather unnecessary as he later on admits to having put the mark there:

He had made quite a dent on the beautiful door; he had also, by the way, knocked out the secret mark that he had put there the morning before ..... "Of course there is a mark," said Gandalf. "I put it there myself."

All references: the Hobbit, Chapter 1, an Unexpected Party

I don't know much about Tolkien stuff, but yesterday I came across an article on wikipedia about "Old English" and it had an image about the runic alphabet and the "f" one says something about "wealth", maybe it has to do with that? The movie version, but I don't know about the books.

I fairly believe that the F belonging on Bilbo's door would either mean "Friend" as in The Lord of the Rings, to answer the Riddle to Enter Moria, the old abandoned mines. Or it would mean Fundin, the Father of Dwalin and Balin, accompanied by Thorin Oakensheild.

Runic f in viking runes is fehu meaning wealth,abundance,energy,foresight,fertility or creation.
But as in the film the dwarves arrive and expect to be fed i'd go for abundance.