If the time spent by one operator is obvious unreasonable, please feedback to the MindSpore Community. If there is no significant difference in precision between float16 and float32 for the same operator, please use float16 to optimize the performance. Step 1: Jump to the Operator Time Consumption Rank tab to see how much time each operator takes during training, focusing on the top-ranked operators. If this phase takes a long time, it is recommended that users analyze it in the following steps: This phase mainly refers to the execution time of forward and reverse operators in the network. Users can reference Optimizing the Data Processing and try to optimize the data processing performance. Principles of judgment can be found in the Performance Profiling page. Step 3:Please jump to the data processing tab on the data preparation details page, observe the inter-operator queue, and determine which operation has a performance bottleneck in the data processing. Otherwise it indicates that the data processing process is the performance bottleneck, please refer to Step 3 to continue to locate which operation of data processing has performance problems. If none of the size in the queue is 0, it indicates that the process by which training data is sent from host to device is a performance bottleneck, please feedback to the MindSpore Community. Step 2: See how the size curve changes in the host queue. If there is 0 in the data queue, go to Step 2. If it is confirmed that the GetNext operators take a long time, please feedback to the MindSpore Community. Based on daily tuning experience, the high probability of this is that GetNext operators are time-consuming, and users can go to the Operator Time Consumption Rank page to view GetNext operators in the AICPU tab. If size is not always 0 in the data queue, it indicates that the data preparation process is not the bottleneck. Step 1:Please jump to the step interval tab on the data preparation details page to see how the size curve changes in the data queue. Computational Graph Visualization Design.Overall Design of Training Visualization.Accuracy Problem Locating and Optimization Guide.Training Optimization Process Visualization.Migrating From Third Party Frameworks With MindConverter.Use Mindoptimizer to Tune Hyperparameters.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |