on LLMs, NLP, RWKV

4 min read

Finetuning RWKV 14bn with QLORA in 4Bit

It was surprisingly easy to get this working, and I think that's a good thing. First I looked at existing LORA implementations of RWKV which I discovered from the very helpful RWKV Discord. The link I found in the discord landed me at "How to Train Your Raven", shout out…