A hands-on writing workshop, titled “Strengths, Weaknesses and Confusion: AI-Assisted Learning in the Classroom”, featuring Mr. Graham Oliver, Adjunct Instructor of the Academic Writing Education Center at National Taiwan University, was organized by the EMI Resource Center at National Chengchi University on April 25th 2024 (Thursday). This event, led by Professor Siaw-Fong Chung, the center director, was held in Dah Hsian Lecture Hall (face-to-face) and Luo Jialun Lecture Hall (through live streaming), Dah Hsian Library on the NCCU Zhinan campus, providing an insightful account of the advantages and limitations of using large language models (LLMs), such as ChatGPT, in English writing. Interaction was facilitated through QR codes, enabling participants to interact with the speaker by raising questions. The speaker further interacted in person with participants in both lecture halls, further enhancing their understanding and application of the topic.

Instead of the commonly-known term “AI”, which he regarded as overly broad and confusing, Mr. Oliver preferred a more precise one “large language models” (LLMs) when referring to programs such as ChatGPT, Google Gemini and Claude. Along with the definition, Mr. Oliver started the workshop with clear focuses: when to use LLMs, when not to, and what to watch out for while using them. He also proposed several metaphors describing LLMs: first, coffee – whether LLMs are good to use depends very much on how we are using them; second, a grain of salt – we need to be careful believing what we are told by LLMs; third, calculator – just like a calculator, LLMs are a useful and convenient tool in our daily life, but we need a lot of foundational knowledge to use them well; fourth, a frenemy editor – most of the time LLMs give us good advice, but in some situations they may not be so constructive as we imagine. Mr. Oliver further cited the word “hallucination” to highlight the possibility of the emergence of untrue, while true on the surface, information by LLMs.

Mr. Oliver emphasized some considerations when we need to make use of LLMs in writing. Comparing common models like ChatGPT and Google Gemini, he concluded that the former tends to produce works of a slightly better quality while the latter tends to better follow our instructions. When it comes to what to include in the prompts, details about the target audience, the spoken or written mode, the tone and the context have to be taken into account. Throughout the entire workshop, Mr. Oliver invited the audience to engage in various tasks and to share their production, through making adjustments to the prompts in terms of writer’s gender, age, and job title, etc., purposefully to see how the results differed. During the comparisons of production in various stages, Mr. Oliver repeatedly reminded the listeners to become active users, instead of passive ones – saving especially good sentences and discarding those not working from later versions. Great differences in writing products may emerge once adjustments to instructions and guidance are administered.

Besides introducing means to achieve better writing, Mr. Oliver, on the other hand, discussed some pitfalls of LLMs to improve one’s production. For example, ChatGPT may be a suitable editor, but may not be a good teacher: it could make changes to language errors, but might not be able to explain the principles behind its edits consistently. Therefore, Mr. Oliver deduced that good proofreading skills are indispensable when one decides to rely on LLMs to produce writing. In addition, LLMs are found to exhibit overuse of grandiose and unnatural vocabulary in almost every single sentence, repetition of the same long sentence structures throughout a discourse (although all grammatically correct) and similarities across a variety of topics.

LLMs, according to Mr. Oliver, are better in improving the structure (such as introduction and conclusion), instead of the content of written works – superficial, incorrect, misleading and even biased information might arise. Some members in the audience expressed having noticed such a feature as well when they made changes in writer’s gender, pronouns or adjectives in their instructions to LLMs – a result showing more professionalism was exhibited when a participating professor changed the identity of the writer from a female researcher to a male one. This was not surprising, responded by Mr. Oliver. Prejudiced content is prevalent online, so bias in the generation of LLMs may not be surprising. Even worse situations in content might occur in ethics and sourcing: accidental plagiarism might be committed since LLMs can lift exact sentences from various sources online without creator’s consent and user’s awareness. The more specific the writing topic, the higher the possibility of this danger may arise. To avert this problem, Mr. Oliver suggested students provide more personalized content and details, especially the ones about themselves, or else overly similar writing is produced for everybody, resulting in overly similar writing being produced for everybody.

Despite potential weaknesses and imperfections which might help us in some situations but hurt us in others, Mr. Oliver felt optimistic about the use of LLMs in the future if we can become active editors, and concluded the workshop with a couple of tips and tricks when deploying LLMs in a helpful way to achieve our writing goals. First, we should build upon what we tell LLMs in a single session, through saving the good results generated (probably around 10–20%) and pasting them each time we start a new session of using LLMs. Second, we have to notice that current LLMs can only do solid close analysis of very widely discussed texts; our effort might be reduced if LLMs are required to do specific tasks as well as process both academic analyses and personal reflection. Third, we have to be accountable to see and critique flaws from LLMs. Fourth, we might consider other LLM-powered tools, namely Elicit, Bing and Semantic Scholar, in addition to common ones like ChatGPT. Finally, LLMs might be a good means to provide subtleties, nuances and appropriateness in the translations of some Chinese phrases and idioms, compared to commonly-used sources like Google Translate.

We express our greatest appreciation for Mr. Oliver’s stimulating sharing, which let us, as teachers and students, understand what prerequisites are involved in using LLMs in writing and in the classroom. On May 30th 2024 (Thursday), Mr. Oliver will conduct another hands-on writing workshop in NCCU on the strategies of peer- and self-editing, along with an introduction of Writefull, an online software application which provides guidance on editing, by the EMI Resource Center (link for registration: https://moltke.nccu.edu.tw/Registration/registration.do?action=conferenceInfo&conferenceID=X22917). We look forward to his next workshop!