Åä¾°

½üÄêÀ´£¬Ëæ×ÅÈ˹¤ÖÇÄÜÊÖÒÕµÄѸÃÍÉú³¤£¬»úеѧϰģ×ӵĹæÄ£·ºÆð³ö±¬·¢Ê½ÔöÌí̬ÊÆ¡£ÓÈÆäÔÚÉî¶ÈѧϰÁìÓò£¬Ä£×Ó¹æÄ£µÄÏÔÖøÌá¸ß³ÉΪÍƶ¯Öî¶àÊÖÒÕÍ»ÆÆÓëÐÔÄÜÌáÉýµÄÒªº¦ÒªËØ¡£µ±Ä£×Ó²ÎÊýÊýÄ¿µÖ´ïÍòÒÚ¼¶±ðʱ£¬ÊÜÏÞÓÚµ¥Ì¨×°±¸µÄÎïÀí×ÊÔ´¶øÎÞ·¨ÊµÏÖµ¥»úѵÁ·¡£Îª´Ë£¬ÂþÑÜʽѵÁ·ÊÖÒÕÓ¦Ô˶øÉú£¬²¢Ñ¸ËÙ³ÉΪѵÁ·³¬´ó¹æģģ×ӵĽ¹µãÊֶΡ£ÂþÑÜʽѵÁ·Äܹ»ÏÔÖøÌá¸ßѵÁ·Ð§ÂÊ£¬Í»ÆƵ¥»úÄÚ´æºÍÅÌËãÄÜÁ¦µÄÆ¿¾±¡£ÈÚºÏÄ£×Ó²¢ÐÐÓëÊý¾Ý²¢ÐеÈÊÖÒÕµÄÂþÑÜʽ»ìÏý²¢ÐÐѵÁ·Õ½ÂÔ³ÉΪѵÁ·³¬´ó¹æÄ£´óÓïÑÔÄ£×ÓµÄÖ÷ÒªÊÖÒÕ;¾¶£¬¸ÃÒªÁì²»µ«ÄÜÓÐÓÃʹÓôó¹æÄ£Ó²¼þ×ÊÔ´£¬»¹ÄÜÔÚÈ·±£ÑµÁ·ÎȹÌÐÔºÍÄ£×ÓÐÔÄܵÄÌõ¼þÏÂ£¬ÊµÏÖ´óÓïÑÔÄ£×ӵĸßЧѵÁ·Óë°²ÅÅ¡£

Ò»¡¢ÍøÂç¹ÊÕÏ

ÂþÑÜʽѵÁ·ÊÇÖ¸ÔÚ¶à¸öÅÌËã½ÚµãÉÏЭͬ×÷Òµ£¬ÅäºÏÍê³É»úеѧϰģ×ÓµÄѵÁ·Àú³Ì¡£Í¨¹ý½«ÑµÁ·Ê¹Ãü²ð·ÖÖÁ¶à¸ö×°±¸Éϲ¢ÐÐÖ´ÐÐ£¬²»µ«ÄܺÏÀí·ÖÅÉÅÌËãºÍ´æ´¢×ÊÔ´£¬Ò²ÄÜÏÔÖøÌáÉýÑ·üçٶÈ£¬²¢ÇÒÄܹ»´¦Öóͷ£¸ü´ó¹æÄ£µÄÊý¾ÝºÍÄ£×Ó¡£

È»¶ø£¬ÑµÁ·Àú³ÌÖзºÆð¹ÊÕϵĿÉÄÜÐÔ»áËæ×ÅѵÁ·¹æÄ£ºÍÒ»Á¬Ê±¼äµÄÔöÌí¶øÉý¸ß¡£Ò»µ©±¬·¢¹ÊÕÏ£¬½«Ê¹ËùÓмÓÈëµÄ×°±¸´¦ÓÚÏÐÖÃ״̬£¬Ö±ÖÁ¹ÊÕÏ×°±¸»Ö¸´Õý³££¬µ¼Ö´ó×ÚËãÁ¦ÎÞ·¨»ñµÃ³ä·ÖʹÓá£À´×ÔMeta¡¢HuggingFaceºÍLAIONµÄÍŶӾù±¨¸æÁËÔÚѵÁ·´óÐÍÄ£×ÓʱÒòʧ°Üµ¼ÖÂʹÓÃÂÊÑÏÖØϽµµÄÇéÐΡ£

ƾ֤MetaÍŶÓѵÁ·Llama3.1µÄ±¨¸æ£¬ÔÚѵÁ·Ê±´úƽ¾ùÿÈý¸öСʱ¾Í»á±¬·¢Ò»´Î¹ÊÕÏ£¬ÆäÖÐÔ¼8.4%µÄ¹ÊÕÏÊÇÓÉÍøÂç×°±¸¹ÊÕϵ¼ÖµÄ¡£²î±ðÓÚÆäËûÓÉÈí¼þÒý·¢µÄ¹ÊÕÏ£¬ÍøÂç×°±¸¹ÊÕÏ´ó¶àΪӲ¼þ¹ÊÕÏ£¬ÀýÈçÍøÂçÏßÀ»òÊÇÍø¿¨¹âÄ
£¿éµÈÓ²¼þ×°±¸·ºÆð¹ÊÕÏ¡£Ïà½ÏÓÚÈí¼þ¹ÊÕÏ£¬Ó²¼þ¹ÊÕϵĴ¦Öóͷ£ÄѶȴ󡢻ָ´Ê±¼ä³¤£¬¿ÉÄÜÔì³É¸ü´óµÄËãÁ¦Ëðʧ¡£

½»Á÷»ú×°±¸¼äÁ´Â·¹ÊÕÏÏÖÔÚÒÑÄܹ»ÊµÏÖºÁÃ뼶´¦Öóͷ£ÓëÊÕÁ²£¬µ«¶Ë²àÍøÂç×°±¸¹ÊÕÏÏÖÔÚÉÐÎÞ½ÏΪ¸ßЧµÄ´¦Öóͷ£¼Æ»®¡£¶Ë²à×°±¸¹ÊÕÏÖ÷Òª°üÀ¨·þÎñÆ÷Íø¿¨¹ÊÕÏ¡¢·þÎñÆ÷Óë½»Á÷»úµÄÅþÁ¬ÏßÀÂËð»µÒÔ¼°ÏßÀ¹âÄ
£¿é¹ÊÕϵÈ¡£Ëæ×ÅÅÌË㼯Ⱥ¹æÄ£Ò»Á¬À©´ó£¬¶Ë²àÍøÂç×°±¸¹ÊÕÏÒѳÉΪؽ´ý½â¾öµÄÎÊÌâ¡£

¶þ¡¢¹ÊÕÏ´¦Öóͷ£

ÏÖÔÚ£¬³ýÈí¼þ¹ÊÕÏÍâ£¬Ó²¼þ¹ÊÕϵĴ¦Öóͷ£Õ½ÂÔͨ³£²»Çø·ÖÏêϸµÄ¹ÊÕÏÀàÐÍ£¬±¬·¢Ó²¼þ¹ÊÕÏʱ£¬Í¨³£½ÓÄÉÏàͬµÄ¼Æ»®¾ÙÐд¦Öóͷ£¡£

¼ì²éµã£¨Checkpoint£©ÊÇÒ»ÖÖ³£¼ûµÄ³¤ÆÚ»¯»úÖÆ£¬ÓÃÓÚÉúÑÄѵÁ·½ø¶È¡£ÔÚ¹ÊÕϱ¬·¢ºó£¬ÏµÍ³»áÁ¬Ã¦¶Ô¹ÊÕÏ×°±¸¾ÙÐÐÐÞ¸´£¬»òÕß½«¹ÊÕÏ×°±¸´Ó¼¯ÈºÖÐÌÞ³ý£¬È»ºó´Ó×î½üµÄ¼ì²éµã¼ÓÔØÄ£×Ó²¢¼ÌÐøѵÁ·¡£ÏÖÔÚ£¬Ðí¶àÓÅ»¯Õ½ÂÔΧÈƼì²éµã»úÖÆÕö¿ª£º

1.EasyCkpt ½ÓÄÉÒì²½»¯¡¢ÌõÀí»¯µÄÉúÑÄ·½·¨£¬ÍŽáÖصþÄ£×Ó¿½±´ÓëÅÌËã¡¢ÍøÂç¸ÐÖªµÄÒì²½´æ´¢Õ½ÂÔ£¬ÊµÏÖÁ˽üºõÁ㿪ÏúµÄÄ£×ÓÉúÑÄ»úÖÆ£¬²¢°ü¹ÜÁË´óÄ£×ÓѵÁ·Àú³ÌÖÐÄ£×ÓÉúÑÄÓë»Ö¸´µÄ¾«¶ÈÎÞËð¡£

2.Gemini Äܹ»½« Checkpoint ÉúÑÄÔÚ¾ßÓиü´ó¾ÛºÏ´ø¿íµÄ CPU ÄÚ´æÖÐ£¬²¢Í¨¹ýһϵÁмƻ®ÊµÏÖÁË´óÐÍÄ£×ÓѵÁ·µÄ¿ìËÙ¹ÊÕϻָ´¡£

Ö»¹ÜÔÆÔÆ£¬ÈÔÎÞ·¨ÍêÈ«×èÖ¹Òò¹ÊÕ϶¨Î»ºÍÄ£×ÓѵÁ·ÖØÆôËù´øÀ´µÄËãÁ¦Ëðʧ¡£

ÈßÓàÅÌË㣨Redundant computation£©¿ÉÒÔ×èÖ¹ÖØÐÂÉèÖúÍÖØÐÂÆô¶¯µÄ¿ªÏú£¬ÔÚ¾ÙÐÐÄ£×ÓѵÁ·Ê±Ê¹Óòî±ðµÄ½Úµã¾ÙÐÐÈßÓàÅÌËã¡£µ±Ä³¸ö½Úµã±¬·¢¹ÊÕÏʱ£¬ÆäËû½ÚµãÄܹ»È¡´ú¹ÊÕϽڵã¾ÙÐÐÅÌËã£¬¿ÉÊÇÕâÑùÒýÈëÁËÀο¿µÄÄڴ濪ÏúºÍÅÌË㿪Ïú£¬½øÒ»²½ÔöÌíÁËѵÁ·±¾Ç®¡£

Èý¡¢97¹ú¼ÊÍøÂçµÄ¶Ë²àÁ´Â·¹ÊÕÏÌÓÉú

97¹ú¼ÊÍøÂçµÄ¶Ë²àÁ´Â·¹ÊÕÏÌÓÉúÊÇÒ»ÖÖ»ùÓÚNCCL£¨NVIDIA Collective Communications Library£©ÊµÏֵĶ˲àÁ´Â·¹ÊÕÏ¿ìËÙ¹æ±Ü¼Æ»®£¬Äܹ»ÔÚÉϲãѵÁ·¿ò¼ÜÎÞ¸ÐÖªµÄÇéÐÎÏÂ£¬×Ô¶¯¶Ô¶Ë²àÁ´Â·¹ÊÕϾÙÐÐʶ±ð²¢´¦Öóͷ££¬ÔÚÄ£×ÓѵÁ·Àú³ÌÖб¬·¢¶Ë²àÁ´Â·¹ÊÕÏʱÄܹ»¼á³ÖѵÁ·²»ÖÐÖ¹£¬ÔÚ¹ÊÕÏ×°±¸Íê³ÉÐÞ¸´ºóѵÁ·ÐÔÄÜ×Ô¶¯»Ö¸´¡£

¶Ë²à¹ÊÕÏÌÓÉú¼Æ»®Í¨¹ýÔÚNCCL¼ÓÈë¹ÊÕÏʶ±ðÒÔ¼°¹ÊÕϹæ±Ü»úÖÆ£¬²¢ÔÚ½¨ÉèÊý¾ÝÁ´Â·Ê±Í¬²½½¨×°±¸·ÝÁ´Â·À´´¦Öóͷ£ÓÉÓÚÓ²¼þ¹ÊÕϵ¼Öµ¥ÌõÁ´Â·ÎÞ·¨Í¨Ñ¶µÄÎÊÌâ¡£

±¸·ÝÁ´Â·ÔÚÖ÷Á´Â·×´Ì¬Õý³£Ê±²»»á¾ÙÐÐÊý¾Ý´«Êä£¬¶Ô´«ÊäЧÂʲ»±¬·¢ÈκÎÓ°Ïì¡£¶øÔÚ·ºÆð¶Ë²àÁ´Â·¹ÊÕÏÖ®ºó£¬Í¨¹ýһϵÁÐÇл»»úÖƽ«Ô­Á´Â·ÉϵÄͨѶʹÃüתÒƵ½±¸·ÝÁ´Â·¾ÙÐд«Êä¡£

97¹ú¼ÊÍøÂçµÄ¶Ë²àÁ´Â·¹ÊÕÏÌÓÉú¼Æ»®¾ßÓÐÒÔÏÂÖ÷ÒªÌØÕ÷£º

1.·ÇÇÖÈëʽ

¸Ã¼Æ»®²»ÊÜÌض¨ÑµÁ·¿ò¼ÜµÄÏÞÖÆ£¬Ò²ÎÞÐè¶ÔÉϲã¿ò¼Ü¾ÙÐÐÐ޸ġ£¹ÊÕϱ¬·¢ºó£¬ÓÉNCCL×ÔÐд¦Öóͷ££¬Éϲã¿ò¼ÜÎÞ¸ÐÖª¡£

2.¿ìËÙÇÒ¿É¿¿

Äܹ»¶Ô¶Ë²àÁ´Â·¹ÊÕϾÙÐкÁÃ뼶µÄʶ±ðÓ붨λ£¬²¢¶Ô¹ÊÕÏÁ´Â·ÉϵÄͨѶʹÃü¾ÙÐпìËÙǨáã£¬¿ÉʵÏÖÃ뼶ÊÕÁ²£¬²¢ÄÜ°ü¹ÜͨѶʹÃü²»ÖÐÖ¹¡£

3.¿É»Ö¸´ÐÔ

ʵʱ¼à²â×°±¸×´Ì¬£¬Èô¹ÊÕÏ×°±¸×´Ì¬»Ö¸´Õý³££¬Äܹ»ÔÚ»Ö¸´µÄ×°±¸ÉÏÖØÐÞͨѶÁ´Â·£¬²¢½«ÆäÖØÐÂÄÉÈëͨѶװ±¸¼¯Èº¡£

97¹ú¼ÊÍøÂçµÄ¶Ë²àÁ´Â·¹ÊÕÏÌÓÉú¼Æ»®¾ßÓÐÒÔϼ¸µãÒªº¦ÊÕÒ棺

1.ÌáÉýϵͳ¿É¿¿ÐÔ

ʹÓö˲à¹ÊÕÏÌÓÉú¼Æ»®Äܹ»ÓÐÓÃ×èÖ¹Òò¶Ë²àÍø¿¨¹âĤÏßÀµÈ×°±¸±¬·¢¹ÊÕ϶øÒýÆðµÄѵÁ·ÓªÒµÖÐÖ¹£¬ÊµÏÖ¶ÏÁ´Ò»Ö±Ñµ£¬Í¨Ñ¶µÄ¿É¿¿ÐÔÌáÉý10±¶¡£

2.½µµÍѵÁ·±¾Ç®

Äܹ»ÓÐÓÃ×èֹѵÁ·¼¯ÈºÒò¶Ë²àÁ´Â·¹ÊÕ϶øµ¼ÖµÄËãÁ¦×ÊÔ´ÆÌÕÅ£¬Í¬Ê±»ùÓÚ×Ô¶¯¹ÊÕϻָ´´¦Öóͷ£»úÖÆÈ·±£ÍøÂç×°±¸ÊµÏÖ×î´óʹÓÃÂÊ£¬´Ó¶øÏÔÖø½µµÍѵÁ·±¾Ç®¡£ ÒÀ¾ÝMetaÐû²¼µÄѵÁ·ÈÕÖ¾£¨Llama 3.1£¬405BÄ£×Ó£©£¬ÔÚΪÆÚ54ÌìµÄԤѵÁ·½×¶Î£¬¹²·ºÆð419´ÎÒâÍâÖÐÖ¹ÇéÐÎ£¬ÆäÖÐÔ¼8.4%ÊÇÓÉÍøÂç¹ÊÕÏËùÖ¡£H100µÄËãÁ¦½¨ÉèÓöÈԼΪ10Ôª/¿¨/Сʱ£¬Èôÿ´ÎÖÐÖ¹»Ö¸´ÐèºÄʱ1Сʱ£¬ÄÇôÍò¿¨¼¯ÈÔÃÄ¿ÎѵÁ·ÒòÍøÂç¹ÊÕÏÔì³ÉµÄËãÁ¦ËðʧԼΪ350ÍòÔª¡£Í¨¹ý¶Ë²àÁ´Â·¹ÊÕÏÌÓÉúÆøÖÆ£¬Ã¿Äê¿ÉÍì»ØÉÏÍòÍòÔªµÄËðʧ¡£

×ܽá

´óÄ£×ÓÂþÑÜʽѵÁ·Àú³ÌÖеĶ˲àÍøÂç¹ÊÕÏ£¬ÕâÊÇÓ°ÏìÄ£×ÓѵÁ·µÄÖØ´ó×è°­¡£¹¹½¨¾«×¼ÇÒ¸ßЧµÄ¹ÊÕÏ´¦Öóͷ£»úÖÆ£¬ÊǸ÷´óÐÍÄ£×ÓѵÁ·ÍŶӵÄÒ»Á¬×·Çó£¬Ò²ÊÇ°ü¹Ü´óÄ£×ÓѵÁ·µÄÖ÷Òª»·½Ú¡£

ͨ¹ýÔÚͨѶ¿âÖÐÌí¼Ó¹ÊÕÏ´¦Öóͷ£Ä
£¿é£¬Äܹ»´ó·ù¶ÈïÔÌ­¶Ë²àÁ´Â·¹ÊÕϵ¼ÖµÄËãÁ¦Ëðʧ¡£Ëæ׿ƻ®µÄÒ»Á¬µü´úÓëÍêÉÆ£¬ÎÒÃǼáÐŶ˲à¹ÊÕÏÌÓÉú¼Æ»®Äܹ»ÔÚ´ó¹æÄ£¼¯ÈºÑµÁ·¹ÊÕÏ´¦Öóͷ£ÖÐÊ©Õ¹¸üΪÖ÷ÒªµÄ×÷ÓÃ£¬Îª¿Í»§´´Ôì¸ü´óµÄ¼ÛÖµ¡£97¹ú¼ÊÍøÂç£¬ÖÂÁ¦×ö×¶Ë²àµÄÍøÂ繩ӦÉÌ£¡