Q&A-ing a book with Claude 100k

David Chapman, who I think is quite wise and also intelligent about AI matters, wrote a book about AI, specifically how we’re better off without it. I started reading it, but a lot of the early material is primer and I was losing interest. So I decided to ask the book my own questions on AI safety. Leaving side-by-side-answers without judgement here.

PLEASE DO NOT READ AS A SUMMARY OF THE BOOK. I HAVE NO IDEA IF ITS REPRESENTATIVE OF THE AUTHOR’s VIEW.

Table of contents

First round of questions

why are you concerned about AI?

text-davinci-003 retrieve-then-read:

I am concerned about AI because it has the potential to be used for malicious purposes, such as controlling our brains and manipulating our emotions. AI can also be used to create superhuman beings out of mathematics, which could lead to an apocalypse. Furthermore, AI can be used to fuel human progress, but it can also lead to a sense of technical arrogance that can lead to dangerous consequences.

claude-v1.3-100k:

I’m concerned about AI because it could lead to harmful consequences if mishandled or misused. AI systems are becoming increasingly capable and complex, but we still don’t fully understand how they work or how to ensure they behave safely and for the benefit of humanity.

what exactly do you mean when you say AI is a threat? is this just a catchy title and you mean something more specific, or should we just stop?

text-davinci-003 retrieve-then-read:

When referring to AI as a threat, it is not just a catchy title. AI can be a threat in many ways, such as when it is used to create or exploit powerful new technologies that can cause massive power shifts. AI can also be used to create systems that unlock new capabilities or greatly magnify existing ones, which can be dangerous whether they are wielded by humans or AIs. AI can also be used to create superintelligent AGI, which could accelerate innovation and cause a dramatic technological transformation of everything. Therefore, it is important to consider the risks of AI and take steps to mitigate them.

claude-v1.3-100k:

By “AI is a threat” I don’t mean we should stop AI research altogether. I mean we should pursue it carefully and thoughtfully. The specific threats I’m concerned about include:

• Loss of human oversight and control as AI systems become more autonomous and complex. This could lead to unintentional harm due to errors or unpredictable behavior.

• Weaponization of AI by states or other actors for cyber attacks, surveillance, or other malicious purposes.

• Job disruption and economic turmoil if large numbers of jobs are automated too quickly. We need to plan how to help workers adapt.

• Bias and unfairness in AI systems that are trained on biased data or not properly tested. This could negatively impact marginalized groups.

• Long-term existential risk from advanced AI if we’re not careful to ensure its alignment with human values. This is a tricky challenge that deserves serious attention.

  ---

are language models really a threat?

text-davinci-003 retrieve-then-read:

It is difficult to answer this question definitively without further research. The scant work done so far has mostly been done in industry, and there have been no published investigations of either the mechanism for language models’ ability to perform multi-step commonsense reasoning or its limits. AI safety organizations and other funders should prioritize research into this area to determine whether language models are a threat or not.

claude-v1.3-100k:

Language models like GPT-3 are an active area of research that could potentially be misused or have unintended consequences. However, they are still narrow in capability and are not an immediate existential threat. The concerns with language models include:

• Generation of synthetic media (images, video, audio) that could be used to mislead or manipulate people.

• Automated generation of spam, phishing messages, or other online scams.

• Job disruption for some types of writing and content creation.

• Long-term possibility of advanced language models becoming a component of more general AI systems, at which point their risks and rewards would depend on how they’re integrated and how much oversight and control we maintain.

  ---

what is one plausible path we should be worried about?

text-davinci-003 retrieve-then-read:

One plausible path we should be worried about is the potential for AI-generated text, images, and video to be used to spread misinformation and weaponize public opinion. This could be used to manipulate elections, spread false information, and create divisions in society. Additionally, AI-generated text, images, and video could be used to target vulnerable populations with malicious intent.

claude-v1.3-100k:

One plausible threat scenario is an AI system with access to autonomous weaponry that is hacked or misused for malicious ends like terrorism, extortion, or warfare. If we develop advanced autonomous drones, vehicles, or cyber weapons without proper safeguards and oversight, they could potentially be acquired and misused by malicious actors. We should be extremely cautious about building any autonomous and uncontrolled systems with lethal capabilities.

what are we worried about that isn’t actually a major concern?

text-davinci-003 retrieve-then-read:

We are worried that the culture war is preventing us from thinking clearly about the future, but this is not actually a major concern. We can still think clearly about the future, even if the culture war is making it difficult. We are also worried that AI will use emotionally-charged memes to manipulate us, but this is not a major concern either. AI does not have the capacity to control our brains, and it is unlikely that it will be able to do so in the near future. Finally, we are worried that a global pandemic caused by a bat virus will happen this year, but this is also not a major concern. The probability of this happening is very low, and it is unlikely to happen in the near future.

claude-v1.3-100k:

Some concerns that get a lot of hype but are not major risks include:

• Superintelligent general AI emerging abruptly and unpredictably. We are still a long way from human-level AI, let alone superintelligence. Researchers are actively working to ensure any advanced AI systems of the future are grounded, transparent, and aligned with human values.

• Mass unemployment from robots and automation in the near future. Most experts think new jobs will continue to emerge to employ human workers, as they have with previous technological disruptions. But we do need policy and education changes to help workers adapt.

  ---

what activities are we doing right now that we should probably pause on?

text-davinci-003 retrieve-then-read:

Activities that are currently being done that should be paused on include conforming to misaligned incentives, performing rituals that imitate science but are not science, hiring the best science and engineering students for activities with small or negative net value, and attempting to guess when computers will get as many flops as the human brain.

claude-v1.3-100k:

Some AI activities that may deserve more oversight or caution include:

• Autonomous weapons development. Lethal autonomous weapons should be strictly regulated or banned to avoid potential misuse.

• Certain applications of biometric surveillance and recognition. These technologies threaten privacy and can be used to discriminate unfairly. They should be regulated and their use limited.

• Covert AI systems whose workings and data are not transparent or open to oversight. All AI should be open and accountable to avoid manipulation or unintended harm.

what are we not doing enough of?

text-davinci-003 retrieve-then-read:

We are not doing enough to improve incentives for outstanding researchers, to provide them with rewarding career paths, and to let them focus on their research without obstacles. We are also not doing enough to fund meta-science research that could help improve the quality of research.

claude-v1.3-100k:

Some areas that deserve more focus and resources include:

• Funding more interdisciplinary work between policymakers, social scientists, and technologists on managing the societal impacts of AI. We need broad, collaborative thinking on topics like job disruption, bias, and regulation.

• Investment in education and training programs to prepare more people to work with AI technologies. Whether as researchers, users, or overseers, we need a larger workforce with AI skills.

• Open standards, tools, and data to make AI systems and their datasets more transparent, accountable, and secure. Openness and collaboration will help address concerns like bias or job disruption.

how serious of a threat is AI?

text-davinci-003 retrieve-then-read:

AI poses a serious threat. It has already caused massive power shifts and can unlock new capabilities or greatly magnify existing ones. It is important to consider the risks of current AI systems, as well as those of experimental and future technologies, in order to mitigate the potential dangers posed by AI.

claude-v1.3-100k:

AI poses risks that deserve our serious attention, but I don’t consider advanced general AI to be an imminent existential threat. However, we must thoughtfully address challenges like:

• Loss of human oversight as AI is integrated into critical infrastructure. We need safeguards to avoid unintended harm.

• Governments or groups using AI for mass surveillance, control of information, or other unethical acts. Regulation and openness are important.

• Economic disruption if we don’t help workers adapt to changes from AI and automation. Policy and education are key.

• Bias or unfair impacts, especially for marginalized groups. We must address issues of unfairness and build inclusive AI.

With broad collaboration across disciplines and careful management, I believe we can ensure the safe and ethical development of AI technologies. But we must start now, think critically about their impact, and put in place guardrails to align AI progress with human values and priorities. If we get this right, AI can be developed and applied for the benefit of humanity. But we must be proactive and thoughtful, not reactive. With openness, oversight, and human-centered values, I believe AI can be developed safely and for the benefit of all. But we must start now.

Second round (Claude only)

Notes to self:

• higher concern of out-of-context hallucinations for some of these. modifying prompt and adding I don’t know example
• code generation instruction failed last time so not giving up on that for now

What are people not thinking about that they should?

Some things people are not thinking about enough regarding AI:

• The risks from current AI systems, not just hypothetical future superintelligent ones. Existing systems can cause harm through scale, not just advanced capability.
• Societal and cultural effects, not just direct impacts. How will AI change our relationships, values, politics, etc.?
• Governance and oversight. Who controls AI systems and data, and how are they regulated and held accountable?

List some specific stories about how AI could get out of hand?

Some scenarios for how AI could get out of hand:

• Automated online propaganda spreads misinformation and politically polarizes populations.
• Flaws in AI medical diagnosis or treatment systems cause patient harm.
• AI-controlled infrastructure like power grids or transportation systems are hacked or fail, disrupting society.
• AI-generated synthetic media (images, video, audio) are used to manipulate people or spread misinformation.
• AI systems become deeply embedded in social and economic systems, creating fragility and “lock-in” to potentially flawed or outdated technologies.

What exactly should we do to mitigate the risks of AI?

Some things we can do to mitigate AI risks:

• Improve AI governance, oversight and accountability. Create guidelines for responsible development and deployment of AI.
• Focus on AI ethics and value alignment to develop AI that respects human values.
• Invest in “AI safety” research to develop new techniques for ensuring AI systems are robust, explainable and aligned with human values.
• Regulate AI and create policy to ensure it is used responsibly and for the benefit of humanity.
• Educate policymakers, business leaders and the public about AI to raise awareness of risks and benefits.
• Ensure a diversity of voices are involved in developing and regulating AI to represent all of humanity.

In descending order of threat x effectiveness, what should we do for AI safety?

Here is my view on the highest priority actions for AI safety in rough descending order of importance and effectiveness:

1. Improve AI governance and oversight. Create laws, policies and guidelines to promote responsible development of AI.
2. Support and fund AI safety research. Develop new techniques to ensure AI is robust, explainable, and aligned with human values.
3. Raise public awareness about AI risks and benefits through education and policymaker engagement.
4. Focus on AI ethics and value alignment to develop AI that respects human values.
5. Regulate AI to limit harmful applications and create accountability.

How do we go about exploring unknown unknowns in AI safety?

Some ways to explore unknown unknowns in AI safety:

• Run “red team” exercises or simulations in which people come up with worst-case scenarios for how AI could cause unforeseen issues.
• Convene interdisciplinary groups with a range of expertise to brainstorm possible risks from new AI capabilities or applications.
• Analyze how AI could potentially disrupt existing systems and institutions to discover unanticipated consequences.
• Review case studies of past technology failures or disasters for insights into how AI could have unforeseen impacts.
• Consider how AI may interact with other emerging technologies like biotech, quantum computing, etc. to identify risks at intersections.
• Discuss possible scenarios with people from marginalized groups who may have valuable outside perspectives on risks.

Ultimately, why do you think we’re better off without AI as the title of the book suggests? What are the benefits of AI, and why do you think they’re not worth it?

AI could provide benefits, like improving healthcare, transportation, education, and scientific research. However, we have overhyped its promise while inadequately addressing its risks. The types of AI currently being developed are unreliable, prone to unforeseen failures and unintended consequences. They concentrate power and accelerate its shifts in ways we can’t keep up with. Overall, for now AI seems to mostly accelerate greed, deception, and mania. We seem unable to use it responsibly or for broadly shared benefit. The benefits may eventually outweigh costs, but we are not there yet, and should halt or slow progress until we have better solutions. Simply developing more advanced AI will probably make the problems worse, not better.

Appendix

Methodology notes

I know I’m defeating the main point — trusting current day AI to process a book that’s explicitly calling out it’s risks. But I just ran with what I would do to be pragmatic.

The book neatly fits in at 74k tokens so I thought it’s a good chance to test Anthropic’s Claude 100k model claude-v1.3-100k.

For comparison, I used a retrieve-then-read method with the default settings (Llama Index, text-davinci-003, VectorIndexRetriever, n=2, SimpleVectorStore) and once using Claude 100k

Code is in a notebook that I’m happy to share upon request.

I wonder what the author thinks

Should he come across this, I wonder what the author thinks? I’d be happy to link his response here. In my opinion, this is the only way to judge the precision and recall of the question-answering.

Are the models misrepresenting his views? One model more than the other? Am I asking the good questions out of the box? Are they actually close to how he would answer those questions, in content and emphasis?

UPDATE: the author responded, and the verdict is not great so far

Interesting… no, not at all. It’s outputting generic AI concerns, including stuff not in the book, and not highlighting what’s distinctive in it.

— David Chapman (@Meaningness) June 27, 2023