The limbic system is a structure in the dogs brain that regulate olfaction, memory, motivations and behaviour (1). It controls how the dog interacts with the world around them and influences behaviour through instinctual responses to stimulus (chasing, digging, jumping, sniffing, eating, etc) and is responsible for how events affect the dog in that environment.
In terms of obedience training and following our instructions, the limbic system engages most when the dog is interested in what is happening in front of them (2). Dogs have specific things that they individually find interesting or rewarding and these things hold the most value to that individual dog or the breed. The limbic system, particularly the hippocampus, is the area of the brain that is in charge of the value of these things. By using this as rewards we can influence our dogs behaviour into obeying what we want through arousing interest. Arousing interest in what you hold improves your dogs learning and gives a positive response to obeying commands.
However if the value of what we have is lower than what the dog is already engaging with, that’s when the limbic system makes the decision not to respond to instructions. By increasing value in what we hold (value of food, toy, engagement, etc) we can override the limbic system (2). The value in something is decided by the dog and depends on the breeds drives. Food is the most consistent motivator for all dogs as all dogs have the predatory drive of consume. By using food that is high value to the individual dog, we can override their limbic system from behaviour they want to be doing to behaviours we want from them. Essentially I can influence my dog behaviour through the limbic system by giving him food while around a stimulus (other dogs, livestock, scent of deer, etc). The difficulty in training comes when the value of what we hold is lower than the drives from the dog.
Inconsistency in training is recorded in the dogs limbic system as well which means that if there are too many discrepancies in the outcome or reinforcement from what the dog expected it gets noted in learning part of the brain, the cerebral cortex (2). So if we trick our dogs one too many times they will quickly catch on because the outcome that they expected did not happen and in tern this also lowers the value of our instructions. (2)
Punishment for dogs natural behaviours or drives can also override the limbic system as punishment causes an avoidance behaviour. The dog will stop following their natural drives because of fear of the outcome by the handler. Interestingly though it does not make the same learning pathways in the brain as the value system because learning is improved when the dogs interested is aroused. Punishment, same as reinforcement, has emotions attached to the limbic system and connections to the hypothalamus which means that the in terms of training and getting the dog to follow our instructions when we override their limbic system, they will do so either happy to listen or scared to disobey.
As an example of this; I’m walking through a field of sheep with my dog on a lead, he wants to chase the sheep (limbic system drive to chase), I want him to walk calmly next to me, so I have two options to override his limbic system, reward (value system) or punishment. I can give him a high frequency of valuable food in my heel position and reward his disengagement from the sheep (positive reinforcement), or I can yank on his collar or add a punishment to force him back into position (positive punishment). As his drive to chase the sheep is very strong, I want him to learn a different behaviour around this stimulus, which is improved by his interest in me as the handler. If I yank him back on the lead it may still work as he will start to obey because of the adverse outcome that follows his actions, but it can also cause a negative association with myself or the sheep because of the connection with the hypothalamus. By having incredibly tasty treats and giving them to him consistently, he starts to look at me more and take his focus off the sheep because the value of what I hold is more important to him and consistently given. The choice of his response to the sheep changes from chase to heel.
References:
When the Nose Doesn’t Know: Canine Olfactory Function Associated With Health, Management, and Potential Links to Microbiota (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5884888#)
Dr Bruce Fogle, The Dogs Mind (Pages 21, 22, 23)