Link Menu Expand (external link) (LinkedIn) Copy Copied
LLMs Unplugged
When
Wed 1-Oct 1:00 PM - 4:00 PM
Location
Level 3 Birch Building, 35 Science Road, Canberra
Contact
Ben Swift
Skills Developed
Evidence EvaluationCritical Analysis
ModeWorkshop ThreadEnabling Technologies

Birch Seminar Room 3.03

About this KNoT

In this hands-on workshop you will train and use your own language model from scratch - with just pen and paper and a bit of dice rolling. Through interactive exercises and guided discussions, you’ll see how language models (even large language models like ChatGPT) are fundamentally information processing machines, turning language inputs into language outputs. The workshop builds from basic principles to more complex applications through an exploration of language modeling as a probabilistic process of predicting “what token comes next” in a sequence, and ends with a poetry slam (for real).

This is one of 4 Cybernetics for social impact KNoTs offered.

Preparation

No pre-reading or prior technical knowledge is required—just bring your curiosity about how information theory shapes our understanding of human and technological communication.

---

However, if you'd like to get a head start on some of the ideas we'll be covering (or if you're just the curious type and you like reading interesting stuff) here are a few free resources.

Andrey Markov & Claude Shannon Counted Letters to Build the First Language-Generation Models covers some of the historical background. Andrey Markov & Claude Shannon Counted Letters to Build the First Language-Generation Models (IEEE Spectrum)

The Dummy's Guide to Modern LLM Sampling covers the "sampling" part of the process in detail, but also has some helpful definitions at the top. Dummy's Guide to Modern Samplers

Brendan O'Conner's slides on N-Gram Language Models are interesting if you want to go further with the maths (although this is much more maths than we'll use - this session really is for everyone).

What You’ll Learn

By the end of this workshop participants will understand how modern language technologies like autocomplete, predictive text, and even AI systems fundamentally operate by learning statistical patterns in language to make predictions about upcoming words or tokens.

How to Complete This KNoT

With your newly constructed language model, in the final part of this workshop you'll use it to participate in a poetry slam - dramatic readings of newly-generated language model poetry. Your participation in this activity is your completion activity.

Submit your Completion Task here:

Submit your completion task for this KNoT via the Qualtrics survey (ANU Login required). You have 10 calendar days after the KNoT to submit it.

Writing your reflection

Most completion tasks ask you to reflect on what you did. For example prompts, sentence starters, and what a strong reflection covers, see Guidance on Reflective Practice. Your completed tasks come together at the end of the semester in the KNoT Completion Evidence.

bars times search arrow-up