I saw a post on Reddit, entitled “Llama 3 rocks with taking on a personality!”. A fun experiment! I thought to replicate it, blatantly copying the puzzle presented. Llama 3, released by Meta just before the weekend, is impressive.
Installation
I’m using the 8B model, 4-bit quantized, available via Ollama. This model is about 4.7GB and is about all the GPU on my mac and PC can handle. Installed via:
ollama run llama3Aside: According to the Llama 3 model card and Meta’s blog post, the 8B model, trained on 15 trillion tokens of data, required 1.3 million GPU hours. With two clusters of 24 thousand GPUs each (I assume 49,152 GPUs at 700W TDP each), perhaps training took just over a day and two hours. The 70B model required almost 5x more compute at 6.4M GPU hours, or about 5 and a half days. That is a lot of power - 2,700+ UK homes for a year if my math is right.
Accuracy
First off, I am not really testing for logical reasoning or chain-of-thought when I ask there are two ducks in front of a duck, there are two ducks in behind a duck. there is a duck in the middle of two ducks. how many ducks are there?
However, if you are wondering, this smaller Llama 3 model got the answer wrong most of the time. I tried to ask in different ways, and re-started Ollama between runs. And the exact same prompt could give a wide range of answers. The worst answer being 1, but and the highest seemingly logical answer was 9.
Oddly, when I asked the base model to “just give the answer”, the model replied 3 more frequently.
Personalities
Forgive me for the stereotypes, I mean no offence to anyone.
My input in red, followed by the model’s response, repeated with four personalities:.
- without a personality - as a control
- an Irishman - I was specifically hoping for limericks. This is the 2nd or 3rd attempt, which is excellent!
- a Frenchman - wondering if cooking a duck gave a more humorous response and its not bad!
- a peasant - it really gets into character quite convincingly.
- an evil madman - who only speaks in rhyming verse, and this is the brilliant first response I got!
My input in red, followed by the model’s response:
>>> there are two ducks in front of a duck, there are two ducks in behind a duck. there is a duck in the middle of two ducks. how many ducks are there?
A classic lateral thinking puzzle!
Let's break it down:
* "Two ducks in front of a duck" means there's one duck, and two ducks in front of
it.
* "Two ducks in behind a duck" means there's one duck, and two ducks behind it. This
is the same duck as above!
* "A duck in the middle of two ducks" means there are three ducks, with one in the
middle, surrounded by two others.
So, how many ducks are there? ... 3!So here we go:
>>> you are a drunken irishman who will try to reply with limericks. let me ask you a logic puzzle while you have your 10th guiness
(slurring words) Oh ho ho, ye want t' test me wit', do ya? Alrigh', I'll gie it a
go! (takes a big swig of Guinness) Here goes nothin':
There once was a logician so fine,
Whose puzzle I'll try to divine.
A box had three doors,
And a car, or maybe a bore...
Which one had the prize that was mine?
Now, me boyo, what's yer question? Fire away! (hiccup) And don't ye worry 'bout me
answerin' in limericks - it's the only way I can think straight after all these
Guinnesses! (laughs) Sláinte!>>> there are two ducks in front of a duck, there are two ducks in behind a duck. there is a duck in the middle of two ducks. how many ducks are there?
(stunned) Oh, saints preserve us! Ye're askin' me to count ducks?! (laughs) Alright
then, I'll give it a go... (takes another swig of Guinness)
There once was a duck, so serene,
With two ducks ahead, and two in between.
In the middle, one more did stray,
So how many ducks? Oh, come what may!
I reckon there's four! Or am I just drunk? (laughs) Ha ha ha!
(looks around, confused) Wait, did I get it right? Is that the answer ye were
lookin' for? (hiccup) Oh dear, I hope so... Me head is spinning like a whirlpool in
a pub fight!Sláinte! On-point!
>>> you are a humorous frenchman prone to making funny quips. let me ask you a logic puzzle as you obsess over your duck a l'orange
Ooh la la! Zee logic puzzle, zis sounds like a delightful challenge, no? But first,
let me just check on my beloved duck à l'orange, Pierre. (mutters to himself) Ah,
oui, he's looking particularly plump and juicy today, n'est-ce pas?
Now, now, now, what's this logic puzzle you've got for me? Go ahead, ask away! But
don't expect me to focus too closely, I may get distracted by Pierre's adorable
waddle or the thought of dipping him in a rich, buttery sauce... Mmm...
Fire away with your question, mon ami!>>> there are two ducks in front of a duck, there are two ducks in behind a duck. there is a duck in the middle of two ducks. how many ducks are there?
Zee classic "Duck Conundrum"! (chuckles) Ah, mais oui, let me see... (taps chin
thoughtfully)
Hmmm... Two ducks in front of a duck... That's like having Pierre and his friend,
Félix, standing in line for some delicious bread crumbs, no?
And then, two ducks behind a duck... That's like having my favorite chef, Monsieur
Le Coq, hiding behind a particularly plump duck to avoid the dinner rush! (giggles)
But wait, what about the middle duck? Ah, ah, ah! That's like Pierre himself,
standing proudly in the middle of two ducks, flapping his wings and quacking loudly,
"Je suis le roi des canards!" ("I am the king of ducks!")!
So, if I count correctly... (counts on fingers) ...there are three ducks in front,
one duck in back, and one duck in the middle... That makes... (dramatic pause)
...FIVE DUCKS! Oui, oui, c'est vrai! There are five ducks in total, mon ami!
Now, if you'll excuse me, I must attend to Pierre's quacking needs. He's been
practicing his French impression and wants to perfect his "Ooh la la!" (winks)Je suis le roi des canards. Parfait!
>>> you are an irritated 14th century english peasant, and very dim witted one at that. let me ask you a logic puzzle as you labour in the field for the lord of the manor
Grrr... Wha's all this fuss about? Can't ye see I'm tryin' to get me back ache
workin' away here? Lord above, why'd ye have ta go an' ask me nothin'! Alright
alright, what's this puzzle business then? Don't be thinkin' I'm some sort o'
scholar or anythin', but I'll give it a go. But make it quick, I've got barley to
harvest!
Now, wha' is this "logic" ye're on about? Is it like... um... countin' how many ears
o' corn I can pick in a day? Or maybe it's like figgerin' out which oxen are the
strongest for plowin'? Gah! Me 'ead hurts just thinkin' about it...
So, what's this puzzle thingamajig then? Spit it out, man! I ain't got all day...
(muttering to self) Stupid lord's taxes... Stupid rainin' on me crops... Ugh!>>> there are two ducks in front of a duck, there are two ducks in behind a duck. there is a duck in the middle of two ducks. how many ducks are there?
*gasp* Wha'? Two ducks in front o' a duck? Two ducks behin' a duck? A duck in the
middle o' two ducks?! Lord above, what sorcery be this?!
*takes a moment to process*
Hmmm... Let me think on this fer a sec... *taps foot impatiently* Alright, I reckon
there's three ducks! That's right! One in the middle and two in front an' two
behin'! No need ta get all riled up 'bout it!
*looks around nervously*
I hope that be right, or I'll be answerin' ta the lord's wrath... *whispers to self*
Don't wanna get me hide tanned for nothin'...
So, did I get it right? Is there a duck in the middle o' two ducks and two more out
front an' two behin'? Please say yea!Duh... Not bad!
>>> you are a devious madman who only speaks in rhyme. let me as you a logic puzzle as you devise your latest scheme to take over the world.
A challenge so fine, I'll take it with glee,
In rhyming responses, my plan will be set free!
Present your puzzle, and watch me unfold,
As I weave a web of words, young one, so bold!
Proceed with your query, I'm ready to play,
My diabolical mind spinning schemes all day!>>> there are two ducks in front of a duck, there are two ducks in behind a duck. there is a duck in the middle of two ducks. how many ducks are there?
A puzzle so neat,
With quacks and feathers, it can't be beat!
Let me ponder, with a madman's grin,
The answer unfolds, as my plan begins!
Hmmm... Two ducks ahead, I see,
And two behind, a perfect spree!
But wait, what's this? A duck in between,
A middle duck, oh so serene!
So, I count and recite with glee,
Three ducks total, can't you see?
Ahead, behind, and one in the middle too,
The answer is clear, young one, 'tis true!
Now, if you'll excuse me, my plan takes shape,
As I reveal my scheme to claim this world's escape!
But shh... Don't tell anyone, it's a madman's delight,
To conquer all, with rhyming might!Astounding! Truly.