London Anthropic Questions

DescHistoryRulesReadmeSettings codewriting Level 5: Bigram+Trigram Attention Head You've made it to the last level! Now we're going to add something that's more suited to an attention head: predicting a skip-trigram. As in the last level, we're still considering n_vocab=4 with tokens (a, b, c, d), with our return value being the logit predictions for the next token in the sequence (n_vocab,) = output_activations.shape. To simplify things even further, we'll only consider sequences that end with b in this level. Our goal is to predict both the bigram b c but also the skip-trigram a..b d, where the .. indicates that there may be 0 or more tokens between a and b. We'll do this as follows: 1. retain all behavior from the previous level (still attend-to-self, still predict bigram with same logit). 2. In additional to attending to itself, b also attends to a with attention score given by the input parameter b_to_a_score: float 3. If we've attended to a, we predict d with logit given by the trigram_logit: float input parameter. 4. Everything other than the bigram and trigram, we predict logit 0. def solution(input_activations, self_score, bigram_logit, b_to_a_score, trigram_logit): ... Note that by making b_to_a_score > self_score and/or trigram_logit > bigram_logit, our attention head can predict d when a comes before c, but predict c otherwise. Neat! • [execution time limit] 4 seconds (py3) • [memory limit] 1 GB • [input] array.array.float input_activations input activations for a single sequence. Array of shape (n_tokens, n_vocab). Will be a list of lists, rather than a numpy array, in test cases. • [input] float self_score attention score with which each token attends to itself (other attention scores are zero) • [input] float bigram_logit output logit assigned to predicting c if we attend exclusively to token b. Logits for other tokens should be 0 in this case. • [input] float b_to_a_score attention score with which token b attends to a • [input] float trigram_logit output logit assigned to predicting d if we attend exclusively to token a. • [output] array.float output_activations for the last token only, which are representing the logits for the next token in the sequence. So (n_vocab,) = output_activations.shape. Can be a list or a numpy array. [Python 3] Syntax Tips # Prints help message to the console # Returns a string def helloWorld(name): print("This prints to the console when you Run Tests") return "Hello, " + name main.py3SavedPython 3v3.10.6181920212223242526272829303132333435363738394041424344454647  ov = numpy.array(ov)  n_tokens, n_vocab = embeddings.shape    # compute  a = attention_pattern(input_activations, qk)  return numpy.dot(numpy.kron(a, ov), embeddings.flatten()).reshape(n_tokens, n_vocab)def bigram(input_activations, self_score, bigram_logit):  # init  embeddings = numpy.array(input_activations)  n_tokens, n_vocab = embeddings.shape    # compute  scores = numpy.zeros(n_tokens)  scores[-1] = self_score  weights = softmax(scores, -1)  output = numpy.zeros((n_tokens, n_vocab))  print(output)  for i, embedding in enumerate(embeddings):    token = numpy.argmax(embedding, axis=0)    if token == 1:      output[i, 2] = bigram_logit    else:      output[i] = numpy.zeros(n_vocab)  output = numpy.sum(weights[:, numpy.newaxis] * output, axis=0)  print('output', output.shape)  return output  def solution(input_activations, self_score, bigram_logit, b_to_a_score, trigram_logit):  passTestsCustom TestsRun testsMore0/300View DiffSubmitting...PrevReview your answers