How to Shrink a Language Model Without Making it Too Dumb

T2Covered by 1 source · first reported by The Pragmatic Engineer / ByteByteGo / Architecture Weekly at 01 Sept, 15:30 UTC

Models have grown roughly 100-fold in a few years, while consumer graphics memory has roughly doubled. It’s not just a matter of tightening things up to make them fit.

Sources

• T2How to Shrink a Language Model Without Making it Too DumbThe Pragmatic Engineer / ByteByteGo / Architecture Weekly01 Sept, 15:30