Seonglae ChoSummary
- 안되는 이유: residual stream에 따라 token distribution 이 계속 바뀌기 때문에 아예 말도안되는 steram 헛소리 나오는듯
- Replacing strategy RMSnorm 이전이후는 별상관없는듯
Use final token
replace all token to context token
def treensform(self, device=None, model_id='seonglae/yokhal-md', push=None, tag=None): self._load(model_id, device) context_embds = [] def get_activations(module, input, output): generate = output[0].size()[0] == 1 if not generate: for embd in output: context_embds.append(embd) modules = list(self.model.modules()) get_context_hook = modules[-2].register_forward_hook(get_activations) self._inference() print(context_embds) get_context_hook.remove() pad_embedding = modules[2](torch.Tensor([self.tokenizer.pad_token_id]).int())[0] def replace_input(module, input, output): generate = output[0].size()[0] == 1 if not generate: print(output.size()) return torch.stack(context_embds)
Hello guys my name is 송중기 ㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋ What the fuck is this shit [tensor([[ 0.0874, -1.0391, 0.2734, ..., 1.0547, -0.3184, -0.5586], [ 0.5859, 0.5352, -0.3965, ..., 0.3438, 0.6250, -1.1953], [ 0.7148, 0.0481, 0.4375, ..., 0.5391, 0.3125, -0.4512], ..., [-0.3652, 1.1094, -0.7578, ..., -0.6797, 0.1211, -0.0635], [-0.1719, 0.2432, -0.8789, ..., -0.7656, 0.4785, 0.6602], [ 0.2637, -0.1084, -0.9922, ..., 0.0376, 1.6094, -1.1094]], device='cuda:1', dtype=torch.bfloat16), tensor([[ 0.7422, -1.1719, -0.6719, ..., -0.1865, 0.1562, -0.5234], [ 0.7422, -1.1719, -0.6719, ..., -0.1865, 0.1562, -0.5234], [ 0.0874, -1.0391, 0.2734, ..., 1.0547, -0.3184, -0.5586], ..., [-0.1826, -0.3535, 0.1914, ..., 0.5938, 0.4023, 0.1826], [ 0.2949, 0.2773, -0.6211, ..., 0.5859, -0.1260, -1.0312], [ 0.0947, -0.2930, 0.1885, ..., -0.4141, 0.2520, -0.5977]], device='cuda:1', dtype=torch.bfloat16)] torch.Size([2, 7, 2048]) Hello guys my name is increadeddedded reluct , increa dises dises Exacts increa increadeddedded reluct unspeak , increa unspeak , increa unspeak unspeak unspeak unspeak unspeak unspeak unspeak unspeak , increa increa toal toal ù ù ù Walkover increaatience increadeddedded McLaugh , increa increa unspeak , increa increadeddeddeddeddedded reluct unspeak unspeak unspeak , increa unspeak , increa unspeak , increa increa unspeak , increa unspeak , increa unspeak , increa unspeak , increa可以让 shenan What the fuck is to,,,............
압축 안해도 안되는거 보니 distribution이 다르다
without padding final token only
def treensform(self, device=None, model_id='seonglae/yokhal-md', push=None, tag=None): self._load(model_id, device) context_embds = [] def get_activations(module, input, output): generate = output[0].size()[0] == 1 if not generate: embds = [vec[-1] for vec in output] for embd in embds: context_embds.append(embd) modules = list(self.model.modules()) get_context_hook = modules[-2].register_forward_hook(get_activations) self._inference() print(context_embds) get_context_hook.remove() pad_embedding = modules[2](torch.Tensor([self.tokenizer.pad_token_id]).int())[0] def replace_input(module, input, output): generate = output[0].size()[0] == 1 if not generate: print(output.size()) for i, embds in enumerate(output): for j, embd in enumerate(embds): embd[:] = pad_embedding if j == len(embds) - 1: embd[:] = context_embds[i] replace_input_hook = modules[2].register_forward_hook(replace_input) self._inference(texts=['fuck', 'yes']) replace_input_hook.remove()
Hello guys my name is 송중기 ㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋ What the fuck is this shit [tensor([ 0.2637, -0.1084, -0.9922, ..., 0.0376, 1.6094, -1.1094], device='cuda:1', dtype=torch.bfloat16), tensor([ 0.0947, -0.2930, 0.1885, ..., -0.4141, 0.2520, -0.5977], device='cuda:1', dtype=torch.bfloat16)] torch.Size([2, 2, 2048]) fuck increa가 increa yes to to to , robus Walkover ?! , sven Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover
with padding final token only
without padding이랑 같아야할거같지만 조금은 다르다
하지만 전반적으로 비슷한거 보니 구현문제는 아닌듯
def treensform(self, device=None, model_id='seonglae/yokhal-md', push=None, tag=None): self._load(model_id, device) context_embds = [] def get_activations(module, input, output): generate = output[0].size()[0] == 1 if not generate: embds = [vec[-1] for vec in output] for embd in embds: context_embds.append(embd) modules = list(self.model.modules()) get_context_hook = modules[-2].register_forward_hook(get_activations) self._inference() print(context_embds) get_context_hook.remove() pad_embedding = modules[2](torch.Tensor([self.tokenizer.pad_token_id]).int())[0] def replace_input(module, input, output): generate = output[0].size()[0] == 1 if not generate: print(output.size()) for i, embds in enumerate(output): for j, embd in enumerate(embds): embd[:] = pad_embedding if j == len(embds) - 1: embd[:] = context_embds[i] replace_input_hook = modules[2].register_forward_hook(replace_input) self._inference() replace_input_hook.remove()
Hello guys my name is 송중기 ㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋ What the fuck is this shit [tensor([ 0.2637, -0.1084, -0.9922, ..., 0.0376, 1.6094, -1.1094], device='cuda:1', dtype=torch.bfloat16), tensor([ 0.0947, -0.2930, 0.1885, ..., -0.4141, 0.2520, -0.5977], device='cuda:1', dtype=torch.bfloat16)] torch.Size([2, 7, 2048]) Hello guys my name is increa가 increa What the fuck is by by by by by-useruseruseruseruseruseruseruser,, by-useruseruser, sven Walkover by-,,, by----- arxivVề by- , maneu maneu maneu maneu maneu unspeak , dises dises disesĐối with a a little little little nack Walkover Walkover Walkover by by-useruser, sven Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover Walkover
another way
# Hook for retrieve context embedding vector def get_activations(module, input, output): generate = output[0].size()[0] == 1 if not generate: for embd in output: context_embds.append(embd[-1]) # Register a hook and inference for retrieval modules = list(self.model.modules()) get_context_hook = modules[-2].register_forward_hook(get_activations) self._inference(chat=chat) get_context_hook.remove() # Hook for replace input embedding vector pad_embedding = modules[2](torch.Tensor([self.tokenizer.pad_token_id]).int())[0] bos_embedding = modules[2](torch.Tensor([self.tokenizer.bos_token_id]).int())[0] def replace_input(module, input, output): generate = output[0].size()[0] == 1 if not generate: print(output.size()) return torch.stack([torch.stack([bos_embedding.to(output.device), emb.to(output.device)]).to(output.device) for emb in context_embds]).to(output.device) # Register a hook and inference treesformer replace_input_hook = modules[7].register_forward_hook(replace_input) self._inference(chat=chat, texts=['how', 'do']) replace_input_hook.remove()