Prior refrigerators,in the cold winter months, it was common to have a cool box aired to the outside.
Indeed, the idea of bringing cold air to keep food properly refrigerated is good.
The freezer cools to around -2 to -10 F, this for long life storage and if the meat is wrapped no cold burning.
The refrigerator side may run from 36 to about 44 Fahrenheit.
To bring cold air to a zone that needs to have low temperature, the air coming in should be dried to avoid the icing as you may seen in those refrigerators that do not have automatic de-icing.
A fellow I helped with his hydro in Canada, several years ago, had a system where he brought air with a blower into a chamber that had piping to cool the material in the pipe (one system was for the refrigerator, the second was to cool the house in summer).
To do a good job, one needs to have ways to accurately control and limit the air as needed, other wise one may "freeze" the refrigerator section, for this reason the air should go to the freezer section and from there to the refrigerator section.
This way the air coming in should be at lower temperature than 0 Fahrenheit.
For your "Hair-brained idea" as you defined it.
You could use those flexible ducts used for the cloth drier going to the outside ( available in several diameters) on for the intake and a second for the exhaust, two blowers, one to push and the other to suck the air out.
A small temperature detector to operate the fans that at the same time could open a gate to allow the air movement when needed, also another external temperature sensor to decide if the outside temperature is low enough for the "cooling" service and if not, to allow the refrigerator pump to operate and at the same time not allowing the gates and/or the Fans to be working.
The controlling logic is simple to do.
Nando